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1 Introduction 

Stochastic models for queues and their networks are usually described by stochastic pro- 
cesses, in which random events evolve in time. Here, time is continuous or discrete. We 
can view such evolution in backward for each stochastic model. A process for this back- 
ward evolution is referred to as a time reversed process. It also is called a reversed time 
process or simply a reversed process. We refer to the original process as a time forward 
process when it is required to distinguish from the time reversed process. 

It is often greatly helpful to view a stochastic model from two different time directions. 
In particular, if some property is invariant under change of the time directions, that 
is, under time reversal, then we may better understand that property. A concept of 
reversibility is invented for this invariance. 

A successful story about the reversibility in queueing networks and related models is 
presented in the book of Kelly [10]. The reversibility there is used in the sense that the 
time reversed process is stochastically identical with the time forward process, but there 
are many discussions about stochastic models whose time reversal have the same modeling 
features. In this article, we give a unified view to them using the broader concept of the 
reversibility. We first consider some examples. 



(a) Stationary distribution is the state distribution of a stochastic process which 
is independent of time. Because of this definition, the stationary distribution is 
unchanged in time reversal for any continuous or discrete time stationary process, 
where a process is said to be stationary if its distribution is unchanged under time 
shift. Hence, the stationary distribution is reversible for any stationary process. 

(b) Stationary Markov process (continuous or discrete time) is again a stationary 
Markov process under time reversal if it has a stationary distribution. This will be 
detailed in Section 2. In this case, the property that transition probabilities from a 
given state are independent of the past history, which is called Markov property, is 
reversible for a stationary Markov processes. 
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(c) Random walk is a discrete time process with independent increments, and again 
a random walk under time reversal. That is, the time reversed process also has 
independent increments. Hence, the independent increment property is reversible 
for a random walk. A random walk is a special case of a discrete time Markov 
process, but does not have a stationary distribution. 

(d) Queueing model is a system to transform an arrival stream to a departure 
stream. If the queueing model can be reversed in time, the roles of arrival and 
departure streams are exchanged, but input-output structure may be considered to 
be unchanged. Thus, the stream structure is reversible for the queueing model. 

These examples may be too general to get something to be useful, but suggest that 
there may be different levels of reversibility depending on characteristics to be interest 
and on a class of stochastic processes (or models). How can we define reversibility in 
those examples ? Obviously, we need a time reversed process. How can we construct 
it ? Is the stationary distribution required for this ? An answer is unclear even for 
this simple question because of the random walk example (c). To make the problems to 
be concrete, we consider discrete and continuous time Markov processes with countable 
state spaces, which are referred to as discrete and continuous time Markov chains, 
respectively, because they are widely used in queueing and their network applications. 



2 Markov chain and its time reversal 

In this section, we construct time reversed processes for a discrete and continuous time 
Markov chains. We first consider the discrete time case because it is simpler than the 
continuous time case but arguments are similar. A discrete time stochastic process 
{X n ; n = 0, 1, . . .} with S is called a Markov chain if, for any n £ Z + = {0, 1,2,.. .} 
and any i,j,i , . . .,i n -i G S, 

P(X n+ i = j\X = i ,Xx = h, . . . ,X n _i = i n -i,X n = i) = P(X n+ i = j\X n = i). 

If the right-hand side is independent of n, then it is said to have a time homogeneous 
transitions, which is denoted by py. That is, 

Pij = P(X n+1 = j\X n = i), i,j E S. 

This is called a transition probability from state i to state j. Throughout this article, we 
always assume for a Markov chain to be time homogeneous. To exclude trivial exceptions, 
we further assume that there is a path from i to j for each i,j £ S, that is, there are 
states ii, . . . , i n _i £ S for some n > 1 such that 

PiiiPili 2 X • • • X Pi n _ 2 in-iPi n -l3 > 0- (1) 

This condition is meant that any state is commutative with any other state, which is 
called irreducibility. 
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If a probability distribution 7r = {7r(«)} on S, that is, > for all z such that 
Z^ieS' 7r (0 = 1' satisfies 

^(i) = ^2n(i)Pij, j^ S i ( 2 ) 

then it is called a stationary distribution of Markov chain {X n }. Note that, 7r(j) > for 
all j G S by the irreducibility. If Markov chain {X n } has a stationary distribution 7r and 
if X is subject to this tt, then X n has the distribution 7r for each n > 1. Define the time 
shifted process {X~ e ; n = —£, — (£ — 1), . . .} by 

X^ = X n+ ,, n = -£,-(£- 1),.... 

Then, {Xn ;n > m} for each m > —I has the same joint distribution as {X n ; n > 0} 
as long as Xq has the stationary distribution tt, and therefore, by letting i — > oo, Markov 
chain {X n ~°°^;n G Z} is well defined, where Z is the set of all integers. Thus, we can 
consider the Markov chain {X n } to be a process starting from — oo. This allows us to 
define a time reversed process {X n } by 

X n = X_ n , neZ. (3) 



Since P(X_ (ri+1) = j) = w(J), we have 

^(Xq = iq, Xi — ii, . . . , = i n -i, X n = i, X n+ i = j) 

= P(X = i , X_x =ii,..., X_( n _x) = i n -i, X_ n = i, X_( n+1 ) = j) 

= P(X_ (n+ i) = j)F(X = i , X-! = ii, . . . , = 2„_!, X_ n = i|X_ (n+1 ) = j) 



and therefore 



F(X n+1 = j\X = i , Xi =?!,..., X n _i = i n _i,X n = i) 

P(Xo = io, X± — i±, . . . , X n _i = i n -i,X n = i, X n+ i = j] 
P(^o = i>o, X\ = it, . . . , X n ^i = i n -i, X n = i) 

^(j)PjiPiin-l X • ■ ■ X p hif) 



n(i)Pu n -i x • • • x Pilio 

F(X n+1 = j\X n = i). 



Hence, the time reversed process {X n } is also a Markov chain with transition probability 
pij given by 

Pij = ^plpji, i,jeS. (4) 

We have defined the time reversed process {X n } by (3), but we also can define it 
as a Markov chain with transition probabilities {pij} of (4). Furthermore, for the latter 
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definition, it is not necessary for {vr(i)} to be a probability distribution, but < ir(i) < oo 
for alH G S satisfying (2) is sufficient. This {vr(i)} is called a stationary measure. This 
also answers why a random walk can be reversed in time because ir(i) = a for any constant 
a > is its stationary measure. We refer to the stationary distribution or measure to a 
reference measure (see Section 3 of [14] for related topics). 

It is notable that we can use a reference measure other than a stationary measure. 
For example, we can consider a time reversed process {X no _ n ;0 < n < n } starting at 
X no — io f° r fixed time Uq > and state i$ G S. In this case, the reference measure 
is concentrated on i . It can be shown that this process is again a Markov chain, but 
does not have time homogeneous transitions. Thus, it is different from {X n }. As a 
reference measure, a stationary distribution (or measure) has the excellent feature that 
the transition probabilities have simple form. 

Thus, the choice of a reference measure is crucial for the time reversed process. 
Throughout this article, we take a stationary distribution (a stationary measure for a 
few places) clS 8b reference measure. 

We next consider a continuous time Markov chain. This is quite easy because most 
arguments are parallel to those of a discrete time Markov chain. 

A stochastic process {X(t)} with state space S is called a Markov chain if, for each 
n G Z + , any s > 0, any < ti < . . . < t n < s and any i, j, io, . . . , i n G S, 

F{X{s + t)=j\X(0) = i Q ,X{t 1 ) = i 1 ,...,X{t n ) = i n ,X{s) = i) 

= F(X(s + t)=j\X(s)=i). 

We denote the right-hand side of this equation by Pij(t) if it is independent of s, which 
we always assume. We also assume that, for each i,j G S, there exists t > such that 
Pij(t) > 0, which is called irreducibility. 

Then, it is known that there exist a(i) > and q(i,j) > such that 

lim T Pij{h) = q(i,j), % ^ j, i,j G S, 
Ho n 

lim \{pu{h) - 1) = -a(i), i G S. 
Ho ti 

In words, a(i) is a rate going out from state i, and q(i,j) is a transition rate from i to j 
under the condition that the transition occurs. It is not hard to see that 

and Markov chain {X(t)} is determined by {q(i,j);i,j G S}. For the X(t) to be well 
defined for all t > 0, a certain regularity condition is required (e.g., see Section 2.2 of [4]). 
However, it is always satisfied in queueing applications. 

This continuous time Markov chain is slightly extended by adding dummy transitions 



4 



by letting q(i,i) > 0. In this case, we redefine a(i) as 

In standard text books (e.g., see [4]), q(i,i) is usually defined as q(i,i) = —a(i). Thus, it 
is notable that our q(i, i) is different from them. As a stochastic process, q(i, i) > is not 
meaningful because it does not change the state. However, it is important for describing 
queueing model because it enables to rightly count arriving customers who can not enter 
a system. We refer to q as a transition rate function. 

Since the Markov chain is irreducible, its stationary distribution ir is uniquely deter- 
mined by 

«(j>(j) = ^2ir(i)q(i,j), j e S. (5) 

ies 

This equation is slightly different from (2) of the discrete time Markov chain. This is 
just because transition rate function {q(i,j)} is not a probability distribution for given 
%. Assume that the Markov chain has the stationary distribution n, then all arguments 
for the discrete time Markov chain are valid for the continuous time Markov chain if we 
replace pjj by q(i,j) and make small modifications like (5). Denote the time reversed 
Markov chain by {X(t)}, then, similar to (4), its transition rate function q is given by 

= Mt?Cm)» i,jeS. (6) 
ir{i) 

From this equation and (5), we immediately have the next fact. 

Lemma 2.1 A probability distribution it on S is the stationary distribution of the 
Markov chain with transition rate function q if and only if there exists a nonnegative 
function q on S such that 

n(i)q(i,j) = n(j)q(j,i), i,j e S, (7) 

X>(m) = J>(U), (8) 

jes jes 

A message from this lemma is that the stationary distribution can be find if we can 
guess q from the time reversed model. This idea is extensively used in the book [10], and 
Lemma 2.1 is called Kelly's lemma (e.g., see [6]). 



3 Reversibility in Markov chain 

From now on, we only consider the continuous time Markov chain {X(t)} with state space 
S and transition probabilities of {X(t)} because the corresponding results for a discrete 
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time Markov chain are similarly obtained. From now on, we change the notation for state 
from i,j G S to x,i'eS for convenience of our discussions. 

We aim to define reversibility for this Markov chain. For this, we first consider the 
strongest case that the Markov chain {X(t)} itself is reversible. Assume that it has the 
stationary measure it, and define the time reversed Markov chain {X(t)} with transition 
rates q(x,x'). Then, {X(t)} is stochastically identically with its time reversed process 
{X(t)} if and only if 

q(x, x') — q(x', x), x,x' G S. 

From (6), this condition is equivalent to 

7r(x)q(x, x') = 7r(x')q(x', x), x, x' G S. (9) 

This Markov chain is said to be reversible. It is notable that (9) yields, for fixed x and 
the sequence of the states x,x\, . . . , x n -\, x' G S, 

, q(x,x 1 )q(xi,x 2 ) y~ ■ ■ ■ y~ q{x n - 2 ,x n - 1 )q(x n -i,x') 

7T{x ) = vr(x)— — — -— — -. 

g(a;',a: n _i)g(x n _i,x n _2) x ■•• x q(x 2 , x 1 )q{x 1 , x') 

as long as the denominator in the right-hand side is positive. Hence, we have the stationary 
distribution tt if Y2x<=s 7r ( x ) < 00 ■ Otherwise, this it is a stationary measure. Thus, the 
reversibility enables us to get the stationary distribution. 

A typical example of a reversible Markov chain is a birth-and-death process, whose 
state space S = Z + and the transition probability q(x, x') is positive for nonnegative 
integers x, x' if and only if x' = x + 1, x' — x — 1 > or x = i' = 0. For this Markov 
chain, let 

n ( x ) = n(0) g(MMM)x-x#-l),x) 
v; v q[x,x- l)q{x- l,x-2) x ••• x g(l,0)' ' v ; 

then (9) is satisfied. Hence, the birth-and-death process is reversible. 

However, a Markov chain for a queueing model may not be reversible. In particular, 
this reversibility is too strong for queueing networks. Motivated by this, we introduce 
weaker definitions of reversibility. 

Definition 3.1 (Reversibility) Assume that the Markov chain {X(t)} with transition 
rate function q has the stationary distribution it, and let q be the transition rate function 
of its time reversed process. Let T be a set of disjoint subsets of S x S. If the Markov 
chain satisfies 

7T(x)q(x,x')= < X )Q( X > X ')> ^T, (11) 

(x,x')eC (x,x')eC 

or equivalently, 

E 7r(x)g(x,x')= Yl ^W,^), CeT, (12) 
(x,x')ec (x,x')ec 

then T is said to be reversible for the Markov chain or concerning q. 
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Remark 3.1 (a) Obviously, if T is the family of all singletons {(x, x')} with x, x' £ S, 
then we have (9), and therefore the Markov chain is reversible, (b) If C = C\ X C% for 
Ci, C2 C 5, then the equation (12) can be considered as a partial balance for transitions 
from C\ to C 2 to be balanced with the one from C 2 to C\ (see, e.g., [20]). Thus, (11) may 
be considered as a partial balance. 



We can further weaken the definition of reversibility by using so called test functions. 
Let M. + (S x S) be the set of all nonnegative valued functions on S x S. For a subset Q 
of Ai+(S x S), Q is said to be reversible for the Markov chain if 

Tt(x)q(x,x')g(x,x') = ^ n(x)q(x,x')g(x,x'), g £ Q. (13) 
x,x'es x,x'<=s 

Let Q be the set of indicator functions l((x, x') £ C) for C £ T of Definition 3.1, where 
l(-) is the function of the statement '■' such that it equals 1 if the statement is true and 
otherwise, then (13) is equivalent to (12). Hence, (13) certainly generalizes (12). However, 
we will go to another way to further weaken the reversibility. 



Definition 3.2 (Structure reversibility) Under the same assumption and notations 
of Definition 3.1, decompose q into transition rate functions {q u ; u £ t/} for a finite index 

set U as 



q(x, x') = q u (x, x'), x, x' £ S. 



Then, if there are a set of transition rate functions {q*; u £ £/} and a family {T u ] u £ U} 
of sets of disjoint subsets of S x S such that 

(i) q^(x, x') < q u (x, x') for x, x' £ S and u £ U, 

(ii) T u is reversible concerning q* for u £ C/, that is, for g*(x,x') = — --^-q u (x', x), 

ir(x) 

Yl 7r ( x )9u( x ' x ') = 7r ( x )C( a; ; a;/ ) ! CeT u ,ueU, 

(x,x')ec (x,x>)eC 
or equivalently, 

^ 7r(x)g*(x,x') = ^ tt^OC^'^), C'eTu, ueU, 

(x,x')eC (x,x')GC 



then the Markov chain is said to be reversible in structure with respect to {g*; u £ U} and 
{T u ; u £ U} or simply to be reversible in structure, where {7^; u £ {7} may be dropped if 
7« — {{^j x5;xeS} for all u EU . 



In queueing applications, counting processes of customers are important. Among 
them, Poisson processes are frequently used. We give its definition below. 
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Definition 3.3 (Poisson process) A counting process N(t) is called a Poisson process 
with rate A > if it satisfies the following conditions. 

(i) It has independent increments, that is, for any n > 2 and any time sequence t = < 
t\ < t 2 < . . . < t n , the increments N(te) — 7V(^_i), £ = 1, 2, . . . , n, are independent. 

(ii) It has stationary increments, that is, for any s,t > 0, the distribution of N(s + t) — 
N(s) is independent of s. 

(iii) It is simple, that is, the increments N(t) — N(t—) at time t is at most 1. 

(iv) A = E(7V(1)) < oo. 

It is not hard to see that this definition is equivalent to that the inter-occurrence 
times for counting are independent and identically distributed subject to the exponential 
distribution with mean 1/A. 

We here note two important facts related to reversibility. Let q(x,x') be transition 
rates of the continuous time Markov chain. For A C S x S, let N^t) be the number of 
transitions of this Markov chain in the time interval (0, t] from x to x' satisfying (x, x') G A, 
where, if (x, x) G A, it is counted with rate q(x, x). Na is called a counting process of A. 

Lemma 3.1 The counting process iV^ is the Poisson process with rate A > whose oc- 
currences are independent of the state of Markov chain just before their counting instants 
if and only if 

£ q(x,x') = \, XES, (14) 
x'eAi(x) 

where A\(x) = {x' G S; (x,x') G A}. 

Remark 3.2 The condition (14) can be weaken using a transition rate function q* such 
that < q*(x, x') < q(x, x') instead of the set A. In this case, let ./V* be the process which 
counts transitions due to q*, then iV* is the Poisson process with rate A if and only if 

q*(x, x') = A, x G S. 

x'es 

This condition corresponds with the structure reversibility of Definition 3.2. 

Lemma 3.1 easily follows from the following facts. The sojourn time at a given state 
of the Markov chain has an exponential distribution, the inter-occurrence times of A^^ are 
exponentially distributed, and mutually independent by the Markov property (see [6] for 
a complete proof). 

Let A = {(x',x); (x,x r ) G A}. The transitions in A can be observed as those in A 
under time reversal. Obviously, A and A are different in general, and therefore counting 
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processes Na and are different. A question is when is also a Poisson process. The 
answer is immediate from Lemma 3.1 since the counting process Na concerning the time 
reversed Markov chain {X(t)} is the counting process in the time forward Markov 
chain {X(t)}. 

Lemma 3.2 Assume that the Markov chain {X(t)} has the stationary distribution ir. 
The counting process is the Poisson process with rate A > whose occurrences are 
independent of the state of Markov chain just after their counting instants if and only if 

£ q{x,x') = \, xeS, (15) 

or equivalently, 

Ti{x)q{x' , x) = 7r(x)A, x E S. (16) 

x'eii(i) 

Note that (14) and (15) imply (12) for T = {{x} x A\{x)\ x G S}. Hence, in the view 
of reversibility, T = {{x} x Ai(x);x G S} is reversible for the Markov chain satisfying 
(14) if and only if (15) (or equivalently, (16)) hold. 



4 Queue and reacting system 

We now describe queueing models by a continuous time Markov chain. As a toy example, 
we first consider a single server queue. Assume that customers arrive according to the 
Poisson process with rate A > 0, customers are served in the manner of first-come and 
first-served with independently and exponentially distributed service times with mean 
l//i (/i > 0), which are also independent of the arrival process. This model is called an 
M/M/l queue. 

Let X(t) be the number of customers at time t of the M/M/l queue. Since the 
remaining times to the next arrivals and to the service completions of a customer being 
served are independent and subject to exponential distributions, X(t) is a continuous 
time Markov chain with state space S = {0, 1, . . .}, and its transition rate function q is 
given by 

{A, x' = x + 1, 
/i, x' = x — 1 > 0, x, x' G S. 
0, otherwise, 

Hence, {X(t)} is a special case of a birth-and-death process, and therefore reversible. 
From (10), the stationary measure tt is given by 

tt(x) = ir{0)p x , x G S, 
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where p = X/p, and satisfies (9). Hence, the stationary distribution exists if and only if 
p<l. 

The M/M/l queue is a simple model, but it has key ingredients of reversibility. For 
this, we introduce sets of pairs of states for arrivals and departures. Let 

A = {(x,x + l);x e S}, D = A = {(x + l,x);x G S}, 

and let Na and Np are counting processes for them, respectively. Then, 

q(x, x') — q(x, x + 1) = A, x G S, (17) 

x'£Ai(x) 

ir(x')q(x', x) = 7r(x + l)(?(x + 1, x) = 7r(x)A, (18) 

x'eAi(x) 

where Ai(a;) = {x 1 G 5; (x, x') G A}. Hence, by Lemmas 3.1 and 3.2, both Na and N D are 
the Poisson processes with the same rate A. Also, by these lemmas, Na is independent of 
the past history of the Markov chain while N& is independent of its future history. These 
facts are also valid for the Mj Mj s queue which increases the number of servers in the 
M/M/l queue from one to s since the Markov chain for the queue is also a birth-and-death 
process. These facts are known as Burke's theorem [5]. 

We now closely look at the M/M/l queue and Burke's theorem, then we can see that 
Poisson arrivals and Poisson departures occur as long as (17) and (18) are satisfied, and 
it may not be necessary for the Markov chain {X(t)} to be a birth-and-death process. In 
this situation, we need to have a larger state space S and to suitably choose the set A for 
arrival and the set D for departures. Another issue is that we may not be able to define 
the arrival process a priori because it may be a superposition of departure processes of 
other queues in network applications. 

For resolving these issues, we separate the exogenous arrival process from the queueing 
system, and introduce the following formulation according to [6]. Let S be the state space 
of the queueing system removing the arrival process. Assume the following dynamics. 

(al) A customer under state x departs with rate q^ D \x, x 1 ) changing x to x'. 

(a2) An arriving customer who finds state x G S changes it to x' G S with probability 
p( A \x,x'), where Ei'esP^t 1 -' 1 ') = 1 f° r eacn x ^ S. 

(a3) The system state x changes to x' with rate q®(x, x') without departure. This state 
transition is said to be internal. 

Let A = {{x,x') G S x S;p^(x,x') > 0}, I = {(x,x') G S x S;q®(x,a?) > 0}, 
D = {(x,x') ESx S;q^(x,x') > 0}, and let 

A 1 (x) = {x' G S; (x, x') G A}, h(x) = {x 1 G S; (x, x') G /}, 
D^x) = {x G S; (x,x') G D}. 
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This model is fairly general as a queueing system, but nothing to do with a customer 
arrival process. We refer to this model as a reacting system. 

We fed Poisson arrivals with rate a > to this reacting system, and define transition 
rates by 

q(x,x') = ap {A) (x,x') + g (D) (x,x') + x,x' G S. (19) 

Denote the Markov chain with these transition rates by {X(t)}. This Markov chain 
represents the dynamics given by (al), (a2) and (a3), where all actions except for (al) 
occur with exponentially distributed inter-occurrence times. 

Assume that this Markov chain has the stationary distribution n. Then, the transitions 
rate function of the time reversed Markov chain {X(t)} is given by (6). A question is 
under what conditions {X(t)} represents a reacting system with Poisson arrivals. 

To answer this question, we consider state transitions of the time reversed Markov 
chain. If it describes a reacting system, then its transition rate function q must have the 
following decomposition. 

q(x,x') = Pp (K \x,x') + g (D) (x,x') + x,x' G S, 

for some constant (3 > to be determined later, where p^ A \ and q^ are arrival 
transitions, departure and internal transition rates, respectively, for the time reversed 
model. 

Note that the state transition functions for A, D and / must correspond with those for 
the departures, arrivals and internal transitions of the time reversed model, respectively. 
Hence, (6) yields 

pWfoarO = I^gOV.x), ( x , x >) e A, 
p ir[x) 

q iD \x,x') = ^^«p (A) (x',x), (x,x') € D, 
ir{x) 

q®(x,a/) = ^lq il] (x\x), (x,x') e I. 

7T{X) 

However, {X(t)} represents a reacting system if and only if 

P (A) {x,x') = 1, xeS. 

i'eDi(i) 

This is equivalent to 

n{x')q {D) {x',x) =/3tt{x), xeS. (20) 

i'eDi(i) 

In the queueing literature, this condition is referred to as a quasi-reversibility (see, e.g., 
[6]). Thus, we have the following theorem. 
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Theorem 4.1 Let {X(t)} be the Markov chain with transition rates (19) for the react- 
ing system with Poisson arrivals with rate a. Assume that this Markov chain has the 
stationary distribution n. Then, the following conditions are equivalent. 

(a) {X(t)} is reversible in structure with respect to g( D ) and Ta = {{x} X -Di(x); x E S}. 

(b) {X(t)} is quasi-reversible, that is, (20) holds. 

(c) The departure process Njj from the reacting system is the Poisson process with rate 
f3 which is independent of the future history of the Markov chain {X(t)}. 

Example 4.1 (M/M/l batch service queue) Modify the MjMj 1 queue in such a way 
that a batch of customers depart at the service completion instant subject to the rule that 
all customers depart if a requested batch size is greater than the number of customers 
in system at the moment while the requested number of customers depart otherwise. It 
is assumed that the requested batch sizes are independently and identically distributed 
with a finite mean. We call this model M/M/l batch service queue. This queue does 
not have Poisson departures if all batch departures are counted as departures, but it does 
if we only count full batches, that is, the batches whose sizes meet the requested batch 
sizes. Thus, this reacting system is reversible in structure if full batches are only counted 
as departures. Details on this model can be found in Section 2.6 of [6]. □ 

In Theorem 4.1, we have assumed the Poisson arrivals. It may be questioned what will 
occur if the arrival process is not Poisson. To answer it, we consider an external source to 
produce arrivals as another reacting system that is independent of the original reacting 
system, where departures from the latter system become arrivals to the external source 
but do not cause any state changes of the source. Thus, we have the closed system in 
which the two reacting systems are cyclically connected. If the time reversed model has 
the same structure, it is not hard to see that the external source produces Poisson arrivals 
and departures from the original system are also subject to Poisson. Hence, the Poisson 
arrival assumption is reasonable to consider the reversibility in structure. 

A reacting system with Poisson arrivals can model not only a single node queue but 
also a network of queues if the state space S is appropriately chosen. A problem is whether 
the reversibility enables us to get the stationary distribution. Obviously, this is easier for 
a smaller system. This motivates us to consider reacting systems as nodes of a network 
queue and to construct the stationary distribution of the network from those of reacting 
systems of Poisson arrivals. This program will be executed in Section 7. 

Another question on the reacting system is whether or not we can consider arrival 
processes depending on the state of the reacting system. We solve this problem incorpo- 
rating exogenous arrivals as one of departures suitably extending the state space S. This 
will be done in Section 6. 

Before looking at those problems, we will discuss a concrete example for the quasi- 
reversibility. 



12 



5 Symmetric service 



The examples in Section 4 are reversible in structure, but have no internal state transition. 
In this section, we introduce the reacting system which have internal state transitions. 
For this, we classify customers by types, and split the transition functions by them. 

Let T be the set of customer types, and classify arrivals and departures by customer 
types. We split p( A \ q^ D \ a and (3 into p( A \u, •), q^(-,u), a u and f3 u with u E T, 
respectively. Thus, the transition rate of the Markov chain is changed to 

q(x, x 1 ) = (a u p^ A \u, x, x') + q {p \x, x 1 , u)j + q^(x, x'), x, x' G S. 

Similar changed are made for the time reversed model, and all the arguments are parallel 
for the reversibility. For example, the quasi-reversibility condition (20) is changed to 

n(x')q {D) (x',x,u) = (3 u tt(x) 7 x G S. (21) 

x'6Ai(x) 

To describe service discipline, we assume that there are infinitely many service po- 
sitions numbered by 1,2, . . ., and customers in system are allocated to positions 1 to n 
when n customers are in system. We further assume that each x in the state space S has 
the following form when n customers are in system. 

x = (n, {(u e , w e );£=l,2,...,n}), u t G T, w t G {1, 2, . . .}, 

where ui and wg are the type and remaining service stage, respectively, of a customer 
in position £. Let k u be the service stage of a newly arriving customer of type u, and 
let r] U j be the completion rate of the stage j, that is, the sojourn time at the stage j is 
exponentially distributed with mean l/r] U j. We assume the following dynamics under the 
state x. 

(bl) A type u customer arrives and chooses position i among {1,2, ,...,n + 1} with 
probability 5(i, n + 1), and customers in positions i, i + 1, . . . , n move to i + 1, i + 
2, . . . , n + 1. That is, x is changed to x'\ 

x' = (n + 1, {(«£, Wi))l = 1, 2, . . . , i}, (u, k u ), {(u e , w £ );£ = i + 1, i + 2, . . . , n}). 

(b2) A type Ui customer in position i with Wi = 1 departs with rate 7(2,71)77^1, and 
customers in positions % + 1, i + 2, . . . , n move to i, i + 1, . . . , n — 1. That is, x is 
changed to 1': 

x' = (n - 1, {(u e , Wi);£=l,2,...,i-1}, {(u £ , we); £ = % + 1, % + 2, . . . , n}). 

(b3) A type customer in position i with Wi > 2 decreases its remaining service stage 
by 1 with rate ^{i,n)rj u . Wi1 and customers in positions £ ^ i are unchanged. That 
is, x is changed to x'\ 

x' = (n, {(u £ , wi); i = 1, 2, . . . , i - 1}, (u h Wi - 1), {(u e , w e ); £ = i + 1, i + 2, . . . , n}). 
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Note that (bl), (b2) and (b3) uniquely specify p^ A ', and q®. This reacting system 
is fairly general, but hard to get the stationary distribution for that with Poisson arrivals. 
Thus, we consider the simpler situation assuming that, for each n > 1, 

4>(n)5(£,n)=j(£,n), £ = l,2,...n, (22) 

where 4>{n) = X]™=i7(^' n )- Service discipline satisfying this condition is referred to as 
symmetric service (see, e.g., [10]). This service discipline includes processor sharing 
and preemptive last-come last-served, but can not be first-come and first-served except 
for exponentially distributed service times. 

It is well known that the reacting system with Poisson arrivals and symmetric service, 
which is called a symmetric queue, has the stationary distribution 7r: 



n 

a,, 



vr(n, {(u t , wt);£=l,2,...,n}) = c 1 JJ "* — , l<w t <k 

e=i ^ 



Ul J 

UfWf, 



if it exists, where c is the normalizing constant (e.g., see [6]). Furthermore, the quasi- 
reversibility (21) holds, and 

q®(x,x') = q®(x,x'), x,x' G S. 

Thus, for the symmetric queue, the structure reversibility holds for not only the depar- 
ture transition rate functions {q^;u G T} but also its internal state transition function 
q®. These facts are particularly useful in incorporating symmetric queues as nodes of a 
network. 

In the symmetric service, the service amount of type u customer is the sum of k u inde- 
pendent exponentially distributed random variables with means 1 / r) u k u , 1/t) u Qi u -i), ■ ■ ■ A/Vui- 
Hence, if we let r\ u j = k u ( u , then the service amount distribution of type u customer is 
/c u -order Erlang distribution with mean ( u . Furthermore, if type u is split to finitely many 
sub-types and if each sub-type has finitely many stages, then the amount of work for type 
u customer is subject to the mixture of Erlang distributions. Thus, we can approximate 
any service amount distribution by this finitely many stage model. 



6 Self-reacting system and balanced departure 

We modify the reacting system so that it includes the exogenous source as a part of the 
system, and departures are immediately transferred to arrivals. We allow to have different 
types for arrivals and departures. Let U be the set of types. In this section, u G U may 
represent characteristics other than a type of customer. For example, it may represents 
batch sizes of arrivals and departures and their vectors. This is the reason why we use 
U instead of T for the symmetric queue. We slightly change the dynamics (al) and (a2) 
while keeping (a3). 

(cl) An entity u is released under state x with rate q( D \x,y,u) changing state x to y. 
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(c2) A released entity u is transferred to entity v! under state y with probability r(u, y, v!) 
where 



^r(u,y,u') = 1, ueU,yeS. 



(c3) A transferred entity it' under state y £ S changes the state from y to x' G S with 
probability p^ A \u',y,x'), where Ylx'esP ( u '> Vi x> ) = ^ ^ or eacn u ' ^ U and y E S. 

We assume that (c2) and (c3) instantaneously follows after (cl). Then, we can define 
the transition rates of the Markov chain {X(t)} as 

q(x,x') = £ <&* A) M + Q®M, x,x'eS, (23) 
u,u'eu 

where 

q£s* A) ( x > x ') = ^2q {D) (x,yiU)r(u,y,u')p {A \u',y,x'). 

We refer to the model described by this Markov chain as a self-reacting system. One 
may think of r as a routing probability matrix. 

Assume that the Markov chain {X(t)} of this self-reacting system has the stationary 
distribution ir. By u, we denote the solution of the following traffic equation for each 

yes, 

^2 v{v, u')r(u', y, u) = v{y, u), uEU,y E S. (24) 

u'&J 

The solution v always exists if U is a finite set because {r(u, y, u');u, u' G S} is a transition 
probability matrix for each y G S. However, it is not unique because any multiplication 
with a function of y is again a solution, but it will be uniquely determined under certain 
reversibility (see Remark 6.1 below). 

We define in the same way as in the reacting system, while f, and are 
defined for a > as 

q ( >{x,y,u) = ^—P y >{u,y,x), u G U,x,y G S, 

TX{X) 
TT ( X J 

P (A) (u, y, x) = — rg (D) (x, y,u), u eU,x,y G 3, 

av(y,u) 

r(u, y,u) = — -r(u ,y,u), u,u G U,y G S, 

i/(y,u) 

where we can choose any a > 0, and we will take a suitable one later. Then, the transition 
rates of the time reversed Markov chain {X(t)} can be written as 

q(x,x')= W O 37 '^') + Q^ix^x'), x, x' e S, (25) 
u,u'eu 
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where 

<W ( x i x ') = Y ^ (D) ( x ' ^' M ) f ( M ' u')p {A) (u',y, x') . 
yes 

Hence, the model described by this time reversed Markov chain is a self-reacting system 
with transfer function r if and only if 

Y P (A) (u, y, x') = 1, Y u ') = !> V ^ s - 

x'eS u'eu 

Since the second equation is followed from (24), these conditions are equivalent to 

n(x')q( D \x', x, u) = av(x, u), x G S. (26) 

x'es 

Thus, we have the following theorem. 

Theorem 6.1 Assume that the Markov chain {X(t)} with transition rates given by (23) 
has the stationary distribution n. Then, the following conditions are equivalent. 

(a) The time reversed Markov chain {X(t)} represents a self-reacting system. 

(b) The self-reacting system is reversible in structure with respect to {q^T^ ; u, v! G U}. 

(c) There is a solution v of the traffic equation (24) that satisfies (26). 

Remark 6.1 v is uniquely determined by (26) because the stationary distribution tt is 
given. 

We consider how to construct the stationary distribution from the solution of the 
traffic equation (24). From the stationary equation (5), the stationary distribution tt 
must satisfy 



A x ) = -tt Y ^')q{x', 



a(x) 
K ' x'es 

where a(x) = J2 x , eS q(x, x'). Substituting v of (26) into this equation and using the traffic 
equation (24), we have 



tx{x) = — r Y n ( x ') ( Y Y q[D) f ' u 'Y( u 'i U )P (A) K V' x ) + ? W ( x 'i x ) 

^ ' x'eS yeS u,u'eU 

= -^r( a Y Y u (y^ u ') r ( u '^y^ u )p {A) ( u ^y^ x ) + Y n ^ q{1) ^ x '' 



yeS u,u'eU x'eS 

-f^( a YY u ( y > u ^ piA) ( u > y > x } + Y kW)q®w> x ) 

a ^ x > yeSueu x'es 



Assume that 
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(c4) q^'(x',x) = q^{x\x) for x',x G S, 



and let b{x) = XL'es ( ? < ^(- r ' which must be less than a(x). Then 7r is uniquely deter- 
mined by 

tt(s) = a y2y2u(y,u)p (A) (u,y,x), x G S. (27) 
a[x) — b(x) z — ' z — ' 

v 1 y > yes u&U 

Thus we can construct the stationary distribution n from v in this case. Thus, it only 
remains to verify (26) for a self-reacting system to be reversible in structure. 

Example 6.1 (Balanced departure under symmetric service) Suppose that a self- 
reacting system satisfies the condition (c4). We consider conditions to have (26) for this 
network. Define $ as 

= / \ t~7 \ J2 "(y, u)p (A \u, y,x), xeS. (28) 
a{ x) - bix) t—' ^— ' 

v ' y ' yeS u&J 

We assume that state x is uniquely determined by y and u when entity u is released under 
state x E S. Denote this x by g(y,u), and as 

qW(x,y,u) = ^4l(x = f(y,u)), x,yeS,u£U. (29) 
$(x) 

We refer to this departure rate to be ^-balanced according to the terminology used in [19] 

(see also Example 6.3 below). Then, if J2 x es^( x ) < 00 > we cnoose a — ( XLes ®( x )) > 
and let 

tt(x) = a$( e S. (30) 

It is easy to check that this tt is indeed the stationary distribution and the reversibility 
condition (26) is satisfied. Thus, by Theorem 6.1, if q^ is given by (29) and if (c4) is 
satisfied, then we have the stationary distribution ir of (30) as long as (28) is satisfied. 
This self-reacting system is reversible in structure. □ 

Example 6.2 (Bach movement network) We further specialize the self- reacting sys- 
tem of Example 6.1 assuming that there is no internal state transitions, that is, b(x) = 0. 
Let S = Z™ and U = that is, its state is a n-dimensional vector with nonnega- 

tive integer entries. We assume that q( D \x,y,u) > if and only if y = x — u + , where 
u + = (tti, «2, • • • , « n ) for u = (uq, Mi, ... , u n ) G U . We further assume that r(u, y, v!) > 
if and only if \u\ = \u'\, where \u\ = uo + u\ + . . . + u n . 

This system can be considered as an open queueing network with n nodes and batch 
movements, where the i-th entry of x G S represents the number of customers at node i, 
and the j'-th entry of u G S is a departing batch for j ^ while Uq is a departing batch 
from external source for q^^x, y, u) > 0. Furthermore, this u are transferred to v! under 
each state y not changing their total numbers. It is notable that, if uq = always holds, 
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then U can be reduced to Z n , and the model is considered as a closed network. This 
model is referred to as a queueing network with batch movements. 

We assume that q( D \x, y, u) > only if y = x — u + and p( A >(u, y, x) > only if x = 
y + u + . These assumptions are reasonable for batch movements. Then, q( D \x, y,u) > 
if and only if p( A \u, y, x) > and (29) becomes 

qV(x,y,u) = V{X ~f' U) l(y = x- u+), x,yeS,ueU. (31) 

We show that we can choose an arbitrary $, For this, we need to verify (28). For the 
batch movement network, (28) becomes 

$(x) = — — fix — u + , u)l(x — u + G S), x G S. 

v ' ueu 

However, this automatically holds because 

a ( x ) = < ? (D) ( x '^' n ) r ( n '2/' n ')P (A) ( M ' ) y ) a;/ ) 

x'eS 1 y&S u,u'eU 



&(x 



j-^r v{x — U + , ti)l(l-« + eS), X G S. 



ueu 



Hence, we can choose an arbitrary $ in (31) as long as it is a positive valued function. This 
model is called a linear network in [12]. We refer to it as a batch movement network 
with balanced departure. There are a lot of literature on this class of networks (see, 
e.g., [3, 9, 18]. 

If all batch sizes are unit, then the batch movement network is reduced to the standard 
queueing network with single arrival and departures. □ 



Example 6.3 (State independent routing) We consider a special case of the batch 
movement network with balanced departure of Example 6.2 in which r(u,y,u') is inde- 
pendent of y. This model is said to have independent routing. 

In this network, v(y,u) can be independent of y, so it can be written as ^(y)v{u) for 
an arbitrary positive valued function $ on S and the solution v of the traffic equation 
(24) which is a function on U. Thus, (31) becomes 

g 0» ( X ,y,u) = i {y = x _ tt +) x , y G S , u G U, (32) 

<&[x) 

where \& and $ in (31) can be arbitrary as long as they are positive valued functions. 
Furthermore, if v{u) is of the form HILi W T ^ or P os hive constants W\.W2, ■ ■ ■ , w n , then v 
is included in ty(x — m + )/$(x) since 

v{u) = = . 

J- 1 u(x-u+) 
i=l y ' 
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Thus, (32) is further simplified as 



q (X ' V > U) = $(s) (y = X ~ x, y E S, u E U, (33) 
and the stationary distribution is given by 

n 

n(x) = C$(x) JJwf* , x E S, 

i=l 

where C is the normalizing constant. This model has been widely discussed in the litera- 
ture. For example, Serfozo [19] called it the Whittle network with ^-balanced departure 
intensities (see Theorem 1.48 of [19]). There are a lot of applications of this special case 
to telecommunication networks (e.g., see [2]). □ 



7 Queueing network and product form 

We return to reacting systems. We here use them as nodes of a queueing network with 
Markovian routing, where the external source is included as one of those nodes. Consider 
a queueing network with n nodes, numbered as 1,2, ... ,n. We also have an external 
source as node 0. Let 

J = {0,l,...,n}. 

Each node i is a reacting systems with state space Sj, departure rate q^ D \ internal tran- 
sition rate q( and arrival probability function p[ A \ Let Tj be the set of customer types 
arriving at and departing from node i. We assume the following dynamics. 

(dl) A departure of type Ui E Tj customer from node i is produced under state E Si 
with rate qf\xi,x' i ,Ui) which changes the state Xj to x\ E Si. 

(d2) A type Ui customer departing from node % goes to node j as type Uj customer with 
probability r(iui, juj) independently of everything else. It is assumed that 

^2 ^2 r ( iu i^ u i) = (^ u i) e J x Tj. 

(d3) An arriving type Uj customer at node j under state Xj E Sj changes the sate to 
x'j E Sj with probability pj(uj,Xj,x'j). It is assumed that 

vf ] (Uj, Xj, x'j) = l, {Uj, Xj) E Tj x Sj,j E J. 

(d4) The state Xi E Si at node i changes to x\ E Si with rate qp(xi,x'i) not producing 
any departure. 
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We model this network by a continuous time Markov chain {X(t)}. Its state space S 
is given by 

S = S xS 1 x---S n , 
its transition rate function q is given by 

q(x,x') = J2^2Y1 ^ A) ( x ' x ') + X^^'^)' 

where 



g^ A) (x x') = \ 9 * ( x ^ x i> u ) r ( m ^ v )Pj ( v i x j, x j) Ylk^j l i x k = x'k), J, 



We refer the model described by this transition rate as a network with reacting nodes and 
Markovian routing. 

We aim to construct a time reversed process of this network model. For this, we 
consider the reacting system of node i with Poisson arrivals with rate a.{. We assume: 

(d5) There exist positive {a^} and {fii Ui } such that the Markov chain with transition 
rate qi given by 

Qiix^ii x^) ^ (^iuiPi (^i) Xj , Xj) -\- q^ (xj , Xj , ttj)^ -|- q^ {x-i , Xj) , X{ , Xj G Si , i G J, 
which has the stationary distribution 7Tj for each and 

^2 ^2 n i( X 'Mi ( X ii X ii U i) = X ii X 'i G S it i G J, (34) 

u ) = Piu^iiuijuj), j G J,Uj G Tj. (35) 



Then, similarly to the reacting systems with Poisson arrivals in Section 4, we define 
times reversed reacting system and routing function in the following way. 

~(a)/ /\ 1 TTj {XjJ ( D )/ / \ ' ez Q a T 

% (Xj,Xj,?/j) —Oii u -P^ ^ (itj, Xj , Xj) , Xj,Xj G 5*j, Itj G TJ, 

TTi(Xi) 



~(i)/ /\ _ ^n^i) (i)/„/ \ / '\ t= Q 

TTiiXi) 

r(iUi,jUj) = ^-r(jUj,iUi). 
The following facts can be proved using Lemma 2.1 (see [6] for its proof). 
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Theorem 7.1 If the network with reacting nodes and Markovian routing satisfies (d5), 
that is, each node is quasi-reversible in separation, then it has the stationary distribution 
7r given by 



k( x ) = n^Osi), x = (x ,xi, . . . ,x n ) E S. (36) 

Furthermore, the network structure of this Markov chain is reversible in structure with 
respect to {g-^,, u E T h v e Tj,i,j e J}. 

Example 7.1 Jackson network is well known to have the product form stationary distri- 
bution, provided it is stable. This network is a special case of the network of Theorem 7.1. 
To see this, let S = Z? +1 and let 



( A )^ >> 
Pi {x,x) 



l(x — x') i = 0, 

l(x , i = x i + l>,x' j = x j ,i^j), i^O, 



q W( xx >) = S X z = ' 

1 \ mm(xj, Si)/iil(x- = Xi - 1 > 0, x'j = Xj, i ^ j), 0, 

r(iu,jv) = p i:j . 

Then, the reacting system with p[ A \ and Poisson arrivals is reversible in structure, 
and it is not hard to see that (d4) is satisfied with ai Ui = fii U . for all % and itj. If we 
replace the i-th reacting systems for % ^ in the Jackson network by a reacting systems 
with symmetric service, (d4) is also satisfied. This network is called a Kelly network 
(see [10]). Its special case limited to some symmetric service is called a BCMP network 
(see [1]), where requested service amounts are generally distributed, but any distribution 
can be approximated by the distribution with finitely stages. □ 



There are examples for Theorem 7.1 in other direction. A customer is said to be 
negative if it removes one customer without any other change at its arriving node if any, 
which was introduced by Gelenbe [7]. Such a customer is said to be a negative signal 
if it instantaneously moves around in a network according to a given Markovian routing 
until it meets an empty node or leaves the network. It is known that Theorem 7.1 can 
be applied to the Jackson network with negative signals. In this case, we may have 
dim 7^ Pirn for some i and See [6] for a full story of those networks. 



8 Other stochastic processes 

We have mainly considered a continuous time Markov chain. The reversibility also may 
be considered for other stochastic processes. A discrete time Markov chain is one of them. 
There are much literature for it, but most results are parallel to those of the continuous 
time case. 
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Both Markov chains have discrete state spaces. To directly consider service times, we 
need continuous valued components in the state of system. A good model for this is a 
generalized semi-Markov process, GSMP for short, and its variants such as reallocatable 
GSMP (see, e.g. [15, 16, 17, 11]). Roughly speaking, the states of those processes are the 
pairs of discrete state and positive real numbers, called remaining (or attaining) lifetimes. 
If one of the remaining lifetimes expires, then it causes the transition of the discrete state, 
which may produce new lifetimes. 

Those GSMP's are particularly useful to prove the so called insensitivity, which means 
that the stationary distribution depends on the lifetime distributions only through their 
means. This insensitively holds if and only if certain partial balance hold, which is equiv- 
alent to the reversibility of our sense. Under this condition, we again have the symmetric 
queues and their networks. There are some related results here if there are service inter- 
ruptions which force customers to leave (see, e.g. [11]). However, no new development 
has been made during the recent ten years. 

There are some other stochastic processes related to queueing networks. For exam- 
ple, they are often considered using reflecting random walk or semi- Martingale reflecting 
Brownian motion, SRBM for short. Both are multidimensional and take values in the 
nonnegative orthant. Furthermore, the time reversed process has been studied for a gen- 
eral Markov processes (see, e.g., [8]). It is known that SRBM on a polyhedral domain, 
which includes the nonnegative orthant as a special case, has the product form stationary 
distribution if and only if the so called skew symmetric condition holds, which implies 
that the time reversed process is also SRBM (see Theorem 1.2 of [21]). 

9 Concluding remarks 

As we have seen, available models are limited under the structure reversibility on queues 
and their networks. Those models are still actively applied, but theoretical study is less 
active now. What can we do to gain more applicability ? There are some comments here. 

1. The reversibility has two ingredients. One is how strongly we set up it. In this 
article, we mainly considered this aspect for queueing network processes. Another 
is which class of stochastic processes we take for considering it. We shortly discussed 
this issue in Section 8. If we take an appropriate class, say, not too small but not 
too large, then we may find more tractable models for queueing networks. This 
direction of study seems to be not well developed. 

2. For the reversibility, we have used a stationary distribution (or stationary measure) 
as a reference measure. We may try other reference measures in Definition 3.1. 

3. The existing studies for the reversibility in queues have been intended to get the 
stationary distribution in closed form. This target may be too big, and it may 
be better to have smaller target. For example, tail asymptotic of the stationary 
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distribution is one of them (see, e.g., [13]). This may includes to reconsider the 
definition of the reversibility as suggested in 2. 
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